REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  NO.  0704-0188 


The  public  reporting  burden  for  this  coilection  of  information  is  estimated  to  average  1  hour  per  response,  inciuding  the  time  for  reviewing  instructions, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  compieting  and  reviewing  the  coiiection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  coilection  of  information,  including  suggesstions  for  reducing  this  burden,  to  Washington 
Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Ariington  VA,  22202-4302. 
Respondents  shouid  be  aware  that  notwithstanding  any  other  provision  of  iaw,  no  person  shaii  be  subject  to  any  oenaity  for  failing  to  comply  with  a  coiiection 
of  information  if  it  does  not  dispiay  a  currentiy  vaiid  OMB  controi  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY) 
20-12-2013 


4.  TITLE  AND  SUBTITLE 
A  Real-Time  Linux  for  Multieore  Platforms 


2.  REPORT  TYPE 
Final  Report 


3.  DATES  COVERED  (From  -  To) 
15-Sep-2009  -  14-Sep-2013 


5a.  CONTRACT  NUMBER 
W911NF-09- 1-0535 


5b.  GRANT  NUMBER 


6.  AUTHORS 
James  H.  Anderson 


5e.  PROGRAM  ELEMENT  NUMBER 
611102 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAMES  AND  ADDRESSES 

University  of  North  Carolina  -  Chapel  Hill 
Offiee  of  Sponsored  Researeh 
104  Airport  Drive,  Suite  2200,  CB  #1350 
Chapel  Hill,  NC  27599  -1350 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS 
(ES) 

U.S.  Army  Researeh  Offiee 
P.O.  Box  12211 

Research  Triangle  Park,  NC  27709-2211 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 
ARO 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

55837-CS.72 


12.  DISTRIBUTION  AVAILIBILITY  STATEMENT 
Approved  for  Public  Release;  Distribution  Unlimited 


13.  SUPPLEMENTARY  NOTES 

The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  contmed  as  an  official  Department 
of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


14.  ABSTRACT 

The  objective  of  this  project  is  to  extend  a  prototype  Linux-based  real-time  operating 
system  developed  previously  at  UNC  (under  ARO  support)  to  obtain  a  fully-functional  OS 
for  supporting  real-time  workloads  on  multicore  platforms.  This  system,  called 
LITMUS-RT  (Linux  Testbed  for  Multiprocessor  Scheduling  in  Real-Time  systems),  allows 
different  multiprocessor  real-time  scheduling  and  synchronization  policies  to  be  specified 


T  TXIV/TTT' 


15.  SUBJECT  TERMS 
real-time,  Linux,  multicore 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

15.  NUMBER 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

ABSTRACT 

OF  PAGES 

UU 

UU 

UU 

UU 

James  Anderson 


19b.  TELEPHONE  NUMBER 
919-962-1757 


Standard  Form  298  (Rev  8/98) 
Prescribed  by  ANSI  Std.  Z39. 18 


Report  Title 

A  Real-Time  Linux  for  Multieore  Platforms 

ABSTRACT 

The  objective  of  this  project  is  to  extend  a  prototype  Linux-based  real-time  operating 
system  developed  previously  at  UNC  (under  ARO  support)  to  obtain  a  fully-functional  OS 
for  supporting  real-time  workloads  on  multicore  platforms.  This  system,  called 
LITMUS-RT  (Linux  Testbed  for  Multiprocessor  Scheduling  in  Real-Time  systems),  allows 
different  multiprocessor  real-time  scheduling  and  synchronization  policies  to  be  specified 
as  plugin  components.  LITMUS-RT  is  open-source  software  (available  at 
http  ://www.  litmus-rt.  org) . 

The  pre-existing  LITMUS-RT  implementation  included  plugins  for  several  real-time 
scheduling  and  synchronization  policies.  However,  as  its  name  suggests,  LITMUS-RT  was 
originally  designed  as  a  testbed  for  empirically  comparing  such  policies.  Extending 
LITMUS-RT  so  that  it  can  host  real  applications  will  require  extensions  on  several  fronts. 

These  include:  (i)  new  synchronization  and  scheduling  algorithms  that  are  better  suited  for 
large  multicore  platforms;  (ii)  new  analysis  and  implementation  methods  for  dealing 
with  non-CPU  resources  such  as  graphics  processing  units  (GPUs);  (iii)  support  for  dynamic 
task  behavior  (tasks  can  be  created,  destroyed,  and  their  timing  parameters  altered  at 
runtime);  and  (iv)  mechanisms  for  supporting  precedence  constraints  and  task  suspensions 
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Awards 

Prof.  James  H.  Anderson  was  elected  ACM  Fellow. 

Prof  James  H.  Anderson  was  appointed  W.R.  Kenan  Distinguished  Professor  of  Computer  Science. 

Prof  James  H.  Anderson  was  elected  IEEE  Fellow. 

Prof  Sanjoy  Bamah  was  elected  IEEE  Fellow. 

The  paper  "Making  Shared  Caches  More  Predictable  on  Multicore  Platforms,"  by  B.  Ward,  J.  Herman,  C.  Kenna,  and  J. 
Anderson,  received  an  Outstanding  Paper  Award  at  the  25th  Euromicro  Conference  on  Real-Time  Systems  in  July  2013. 

The  paper  "Fair  Lateness  Scheduling:  Reducing  Maximum  Lateness  in 

G-EDF-like  Scheduling,"  by  J.  Erickson  and  J.  Anderson,  received  the  Best  Paper  Award  at  the  23rd  Euromicro  Conference 
on  Real-Time  Systems  in  July  2012. 

The  paper  "Global  Mixed-Criticality  Scheduling  on  Multiprocessors," 
by  H.  Li  and  S.  Bamah,  received  an  Outstanding  Paper  Award  at  the 
23rd  Euromicro  Conference  on  Real-Time  Systems  in  July  2012. 

The  paper  "The  OMLP  Family  of  Optimal  Multiprocessor  Real-Time  Locking 
Protocols,"  by  B.  Brandenburg  and  J.  Anderson,  appeared  in  a  special 
issue  of  Design  Automation  for  Embedded  Systems  consisting  of  the  best 
papers  from  the  2011  ACM  International  Conference  on  Embedded  Software. 

The  paper  "Globally  Scheduled  Real-Time  Multiprocessor  Systems  with 
GPUs,"  by  G.  Elliott  and  J.  Anderson,  appeared  in  a  special  issue  of 

Real-Time  Systems  consisting  of  the  best  papers  from  the  19th  International  Conference  on  Real-Time  and  Network 
Systems. 

The  paper  "Optimality  Results  for  Multiprocessor  Real-Time  Locking," 

by  B.  Brandenburg  and  J.  Anderson,  received  the  Best  Student  Paper  Award  at  the  3 1st  IEEE  Real-Time  Systems 
Symposium  in  December  2010. 

The  paper  "Supporting  Soft  Real-Time  Parallel  Applications  on  Multiprocessors"  was  accepted  for  a  special  issue  of  the 
Journal  of  Systems  Architecture  on  best  papers  from  the  18th  IEEE  International  Conference  on  Embedded  and  Real-Time 
Computing  Systems  and  Applications. 

The  paper  "An  Optimal  k-Exclusion  Real-Time  Locking  Protocol  Motivated  by  Multi-GPU  Systems,"  by  G.  Elliott  and  J. 
Anderson,  appeared  in  a  special  issue  of  Real-Time  Systems  on  best  papers  from  the  19th  International  Conference  on  Real- 
Time  and  Network  Systems. 

The  paper  "Globally  Scheduled  Real-Time  Multiprocessor  Systems  with  GPUs,"  by  G.  Elliott  and  J.  Anderson,  appeared  in 
a  special  issue  of  Real-Time  Systems  on  best  papers  from  the  18th  International  Conference  on  Real-Time  and  Network 
Systems. 

Bjorn  B.  Brandenburg's  Ph.D.  dissertation  received  three  awards:  the  2012  Dean's  Distinguished  Dissertation  Award  in  the 
area  of  Mathematics,  Physical  Sciences,  and  Engineering  (a  UNC  award),  the  2012  Council  of  Graduate  Schools/ProQuest 
Distinguished  Dissertation  Award  in  Mathematics,  Physical  Sciences  and  Engineering  (a  national  award),  and  the  2012 
EDAA  Outstanding  Dissertation  Award  (an  international  award  in  embedded  systems). 
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Scientific  Progress 


LITMUS-RT  provides  plugins  for  several  real-time  scheduling  and 
synchronization  algorithms.  The  scheduling  algorithms  that  are  provided  can  be  used 
to  host  either  hard  or  soft  real-time  workloads.  In  a  hard  real-time  application, 
task  deadlines  cannot  be  missed.  In  a  soft  real-time  application,  some  deadline 
tardiness  is  permissible.  In  the  definition  of  "soft  real-time"  that  we  have  focused 
on,  deadline  tardiness  is  required  to  be  (provably)  bounded.  While  the  same  scheduling 
algorithm  can  be  used  to  schedule  both  types  of  applications  (hard  and  soft),  different 
analysis  is  required  to  prove  real-time  correctness  for  hard  and  soft  real-time 
applications,  as  zero  tardiness  is  required  in  the  former  case,  but  merely  bounded 
tardiness  in  the  latter.  In  this  project,  we  identified  the  best  scheduling 
and  synchronization  policies  to  use  for  both  hard  and  soft  real-time  systems.  The 
metric  of  comparison  is  schedulability:  we  seek  algorithms  that  can  correctly  schedule 
the  largest  possible  category  of  systems,  with  overheads  considered.  The  phrase 
"correctly  schedule"  has  a  different  interpretation  for  hard  and  soft  real-time  systems 
and  it  implies  that  analysis  exists  that  can  be  used  to  predict  which  systems  are 
schedulable.  (That  is,  it  should  be  possible  to  validate  a  system's  timing 
constraints.) 

The  pre-existing  LITMUS-RT  implementation  provided  support  for  a  variety  of  basic 
scheduling  approaches,  ranging  from  partitioning  approaches  (tasks  are  statically  assigned 
to  processors),  to  global  approaches  (tasks  are  scheduled  from  a  single  global  runqueue), 
to  clustered  approaches  (tasks  are  partitioned  onto  clusters  of  cores,  with  global 
scheduling  used  within  each  cluster).  However,  as  its  name  suggests,  LITMUS-RT  was 
originally  designed  as  a  testbed  that  facilitated  the  empirical  comparison  of  different 
scheduling  and  synchronization  policies.  Our  goal  in  this  project  was  to  extend  LITMUS-RT 
so  that  it  is  a  fully-featured  system  that  can  host  real  applications.  These  extensions 
include:  (i)  new  synchronization  and  scheduling  algorithms  that  are  better  suited  for  large 
multicore  platforms;  (ii)  new  analysis  and  implementation  methods  for  dealing  with 
non-CPU  resources  such  as  graphics  processing  units  (GPUs);  (iii)  support  for  dynamic 
task  behavior  (tasks  can  be  created,  destroyed,  and  their  timing  parameters  altered  at 
runtime);  and  (iv)  mechanisms  for  supporting  precedence  constraints  and  task  suspensions 
(e.g.,  due  to  I/O).  Additionally,  we  have  use  (and  are  using)  LITMUS-RT  to  experiment  with 
real  applications.  Some  efforts  in  this  direction  involving  military  avionics  have  already 
commenced  in  joint  work  with  colleagues  in  industry  and  at  research  labs. 

Significance 

Multicore  platforms  provide  significant  processing  power  that  could  potentially  be 
harnessed  to  support  complex,  resource-demanding  real-time  applications.  However, 
for  this  to  happen,  appropriate  real-time  scheduling  and  synchronization  algorithms 
must  be  devised,  analyzed,  and  implemented  within  a  working  OS.  That  was  precisely 
the  objective  of  this  project. 

LITMUS-RT  has  matured  to  the  point  of  being  a  very  stable  and  useful 

platform.  It  has  a  growing  international  user  base.  It  is  now  one  of  the 

standard  de  facto  platforms  for  experimental  research  in  the  real-time  systems 

research  community.  To  date,  it  has  been  used  by  20-30  research  groups  around  the  world. 

Accomplishments 

We  have  re-based  LITMUS-RT  to  the  latest  Linux  on  (approximately)  an  annual 
bassis.  Currently,  it  is  based  on  Linux  3.10.5. 

In  work  on  mixed-criticality  systems,  we  developed  a  scheduling  framework  for 
supporting  tasks  of  different  criticalities  on  multicore  platforms.  These  are 
systems  in  which  subsystems  of  differing  criticalities  exist:  more  critical 
subsystems  require  more  conservative  provisioning  than  less  critical  ones.  For 
example,  in  avionics  systems,  flight  control  subsystems  are  more  critical  than 
mission  control  subsystems.  For  such  systems,  mixed-criticality  scheduling 
algorithms  have  been  proposed  that  attempt  to  reclaim  system  capacity  lost  to 
worst-case  execution  time  pessimism.  Such  algorithms  seem  to  hold  great  promise 
for  multicore  real-time  systems,  where  such  loss  is  particularly  severe  (due  to 
difficulties  in  predicting  contention  with  respect  to  shared  hardware 


resources  such  as  caches  when  assessing  task  execution  times).  Our  group  was 
actually  first  (as  far  as  we  know,  only)  group  to  propose  and  implement  a 
mixed-criticality  scheduling  framework  for  multicore.  As  in  other  work  on 
mixed-criticality  scheduling  (mostly  uniprocessor  work),  our  framework  assumes 
that  high-criticality  tasks  are  provisioned  conservatively  and  thus  will  generate 
significant  processing  "slack"  at  runtime.  Our  framework  allocates  this  slack  to 
low-criticality  tasks  so  that  their  timing  constraints  can  be  met. 

Using  LITMUS-RT,  we  completed  a  major  study  on  implementation  tradeoffs  affecting 
mixed-criticality  systems  using  this  framework.  The  unique  nature  of  these 
mixed-criticality  scheduling  algorithms  gives  rise  to  a  number  of  major  challenges 
for  the  would-be  implementer.  In  our  study,  we  produced  the  first  ever 
implementation  (in  LITMUS-RT)  of  a  mixed-criticality  scheduling  framework  on 
a  multicore  system.  In  this  study,  we  experimentally  evaluated  design  tradeoffs 
that  arise  when  seeking  to  isolate  tasks  of  different  criticalities  and  to 
maintain  overheads  commensurate  with  a  standard  real-time  OS.  We  also  evaluated 
a  key  property  needed  for  such  a  system  to  be  practical:  that  the  system  be 
robust  to  breaches  of  the  optimistic  execution-time  assumptions  used  in 
mixed-criticality  analysis.  This  work  was  partially  funded  by  AFOSR,  but  given 
its  LITMUS-RT-oriented  focus,  some  of  the  students  involved  were  supported  by 
this  project. 

Throughout  the  project,  we  have  investigated  research  issues  involving 
tardiness  analysis  for  soft  real-time  systems  scheduled  using  global 
real-time  scheduling  algorithms.  We  extended  prior  tardiness  analysis  so  that 
task  execution  times  can  be  specified  stochastically  and  tardiness  bounded  in 
expectation.  This  result  has  important  implications  for  soft-real-time 
applications.  Such  applications  are  typically  provisioned  assuming  average-case, 
rather  than  worst-case,  execution  times.  Prior  analysis  required  that  worst-case 
times  be  known.  We  also  improved  upon  prior  tardiness  analysis  by  developing  new 
analysis  that  allows  lower  tardiness  bounds  to  be  established  than  was  previously 
possible.  We  also  devised  new  tardiness  analysis  that  can  be  applied  in  systems  where 
tasks  may  suspend  due  to  I/O  operations,  have  precedence  constraints  (expressible 
using  directed  acyclic  task  graphs),  and  have  non-preemptive  sections. 

We  also  extended  our  work  on  supporting  graph-specified  precedence  constraints  to 
be  applicable  to  real-time  workloads  that  are  hosted  across  multiple  nodes  in  a 
distributed  system.  We  also  devised  new  analysis  that  identifies  a  "best"  global 
scheduling  algorithm  from  the  perspective  of  tardiness  among  a  family  of  algorithms 
that  from  an  implementation  perspective  are  equivalent  to  the  widely-studied 
global  earliest-deadline-first  (GEDF)  algorithm.  Surprisingly,  we  found  that  GEDF 
itself  is  not  the  best  algorithm  within  this  family.  Additionally,  we  devised  new 
tardiness  analysis  for  task  systems  wherein  tasks  may  be  multi-threaded  and  for 
task  systems  in  which  consecutive  invocations  of  a  task  may  execute  concurrently. 

A  common  theme  in  both  cases  is  allowing  intra-task  parallelism.  We  also  obtained 
new  tardiness  analysis  for  task  systems  wherein  tasks  may  suspend  execution 
to  access  external  devices.  Finally,  we  showed  that  by  splitting  jobs  (i.e., 
task  invocations)  into  sub-jobs,  both  observed  tardiness  and  tardiness  bounds  can 
be  reduced,  even  when  the  additional  overheads  of  splitting  jobs  in  practice  are 
considered  (overheads  increase  because  tasks  are  preempted  more  frequently).  This 
result  follows  because  job  splitting  reduces  task  execution  times  and  both  tardiness 
bounds  and  observed  tardiness  are  greater  when  execution  times  are  greater. 

Using  LITMUS-RT,  we  completed  a  major  experimental  study  that  examines  different 
real-time  scheduling  strategies  on  large  multicore  systems  (in  our  study,  a  24-core 
Intel  platform  was  used).  This  work  showed  that,  when  real  overheads  are 
considered,  partitioned  schedulers  are  better  to  use  for  supporting  hard  real-time 
workloads,  and  clustered  schedulers  are  better  for  soft  real-time  workloads 
(here,  global  scheduling  is  applied  within  clusters  of  four  to  eight  cores). 

We  also  completed  a  major  study  on  the  efficacy  of  "semi-partitioned"  real-time 
scheduling  algorithms.  Such  algorithms  have  been  proposed  as  a  compromise  between 
pure  global  and  partitioned  approaches.  Under  a  semi-partitioned  algorithms,  most 
tasks  are  assigned  to  processors  (like  under  partitioning)  but  a  few  may  migrate 


(like  under  global  scheduling).  This  study  evaluated  the  best  (from  an  analytical 
perspective)  seml-partitloned  algorithms  proposed  to  date  and  compared  them,  in 
terms  of  real-time  schedulability  with  measured  overheads  considered,  to  pure 
partitioning  and  global  approaches.  Semi-partitloned  scheduling  was  actually 
shown  to  be  the  preferred  method  for  some  workloads.  However,  semi-partitloned 
schedulers  require  an  offline  task-assignment  phase,  and  this  makes  it  difficult 
to  apply  such  schedulers  in  systems  where  dynamic  workload  changes  can  occur. 

We  developed  a  number  of  real-time  locking  protocols  that  have  provably  optimal  blocking 

behavior.  These  are  the  first  such  algorithms  to  be  proven  optimal.  This  work 

was  mostly  supported  by  NSF,  but  some  of  the  students  involved  in  this  work  were 

supported  under  this  project.  The  protocols  we  have  devised  include  spin-based 

and  suspension-based  locks  for  mutex  synchronization  and  reader/writer  synchronization, 

and  extensions  to  these  protocols  that  allow  fine-grained  lock  nesting  and 

replicated  resources.  We  have  conducted  numerous  LITMUS-RT  experimental  schedulability 

studies  In  which  these  various  protocols  were  evaluated,  with  real  overheads 

considered.  (Summaries  of  many  of  these  studies  can  be  found  in  Bjorn  Brandenburg's 

Ph.D.  dissertation;  a  good  overview  of  LITMUS-RT  can  be  found  there  as  well.) 

Based  on  this  synchronization  work,  we  tackled  the  problem  of  making  shared 
caches  more  predictable  on  multicore  platforms.  In  safety  critical  domains,  the 
usage  of  multicore  platforms  has  been  hampered  by  problems  due  to  interactions 
across  cores  through  shared  hardware.  The  inability  to  precisely  characterize 
such  interactions  can  lead  to  pessimism  in  worst-case  execution  time  analysis  that 
is  so  great,  the  extra  processing  capacity  of  additional  cores  Is  entirely 
negated.  Our  work  addresses  this  issue  as  it  pertains  to  shared  caches.  Our 
approach  utilizes  an  old  Idea  called  "page  coloring,"  under  which  pages  of 
physical  memory  are  "colored"  so  that  differently-colored  pages  cannot  cause  cache 
conflicts.  We  view  such  colors  as  shared  resources:  the  real-time  locking 
protocols  just  described  are  used  to  enable  each  task  to  lock  Its  needed  colors 
whenever  it  is  invoked.  We  have  implemented  and  evaluated  this  page-coloring  approach 
using  LITMUS-RT.  We  have  since  expanded  our  work  on  predictably  managing  shared 
non-CPU  hardware  resources  like  caches  into  a  major  NSF  project. 

In  work  that  was  partially  funded  by  NSF,  we  produced  an  extension  to  LITMUS-RT 
called  GPU-Sync,  which  is  a  framework  for  managing  graphics  processing  units  (GPUs) 
in  multi-GPU  multicore  real-time  systems.  (GPU-Sync  more  than  doubles  the 
LITMUS-RT  code  base.)  GPUSync  was  designed  with  flexibility,  predictability,  and 
parallelism  in  mind.  Specifically,  It  can  be  applied  under  either  static-  or  dynamic- 
priority  CPU  scheduling;  can  allocate  CPUs/GPUs  on  a  partitioned,  clustered,  or  global 
basis;  provides  flexible  mechanisms  for  allocating  GPUs  to  tasks;  enables  task  state 
to  be  migrated  among  different  GPUs,  with  the  potential  of  breaking  such  state  into 
smaller  "chunks";  provides  migration  cost  predictors  that  determine  when  migrations 
can  be  effective;  enables  a  single  GPUas  different  engines  to  be  accessed  in  parallel; 
properly  supports  GPU-related  interrupt  and  worker  threads  according  to  the  sporadic 
task  model,  even  when  GPU  drivers  are  closed-source;  and  provides  budget  policing  to 
the  extent  possible,  given  that  GPU  access  Is  non-preemptive.  No  prior  real-time  GPU 
management  framework  provides  a  comparable  range  of  features.  GPU-Sync  utilizes  the 
aforementioned  optimal  real-time  locking  protocols  to  arbitrate  access  to  GPU-related 
resources. 

Conclusions 

In  our  algorithmic  research,  we  tackled  fundamental  resource-allocation 
problems  that  need  to  be  addressed  to  enable  both  soft  and  hard  real-time  workloads 
to  be  hosted  on  multicore  platforms.  In  our  OS  and  prototyping  efforts,  we 
showed  that  such  algorithms  are  not  merely  of  theoretical  interest,  but  can 
be  practically  applied. 

Technology  Transfer: 

Our  work  on  mixed-criticallty  systems  was  done  jointly  with  colleagues  at 
Northrop-Grumman  Corp.  Also,  we  did  some  joint  work  on  data-stream  processing 


on  multicore  platforms,  where  soft  real-time  constraints  apply,  with  a  colleague  at 
AT&T  Research.  We  also  did  some  joint  work  with  colleagues  at  a  company  called 
Real-Time  Innovations  on  real-time  computing  on  "many-core"  (10s  to  1000s  of  cores  on 
a  chip)  platforms  that  resulted  in  Phase  1  STTR  funding  from  the  U.S.  Air  Force  Research 
Laboratory.  Our  GPU-related  work  has  recently  resulted  in  new  funding  from  General 
Motors  Corp. 

Technology  Transfer 


